Multi-objective construction of an entire adequate test suite for an EFSM

In this paper we propose a method and a tool to generate test suites from extended finite state machines, accounting for multiple (potentially conflicting) objectives. We aim at maximizing coverage and feasibility of a test suite while minimizing similarity between its test cases and minimizing overall cost. Therefore, we define a multi-objective genetic algorithm that searches for optimal test suites based on four objective functions. In doing so, we create an entire test suite at once as opposed to test cases one at a time. Our approach is evaluated on two different case studies, showing interesting initial results

Combining Static and Dynamic Analyses to Reverse-Engineer Scenario Diagrams

This paper discusses reverse engineering source code to produce UML sequence diagrams, with the aim to aid program comprehension and other software life cycle activities (e.g., verification). As a first step we produce scenario diagrams using the UML sequence diagram notation. We build on previous work, now combining static and dynamic analyses of a Java software, our objective being to obtain a lightweight instrumentation and therefore disturb the software behaviour as little as possible. We extract the control flow graph from the software source code and obtain an execution trace by instrumenting and running the software. Control flow and trace information is represented as models and UML scenario diagram generation becomes a model transformation problem. Our validation shows that we indeed reduce the execution overhead inherent to dynamic analysis, without losing in terms of the quality of the reverse-engineered information, and therefore in terms of the usefulness of the approach (e.g., for program comprehension)

Comparing GUI Functional System Testing with Functional System Logic Testing - An Experiment

The practitioner interested in reducing software verification effort may found herself lost in the many alternative definitions of Graphical User Interface (GUI) testing that exist and their relation to the notion of system testing. One result of these many definitions is that one may end up testi ng twice the same parts of the Software Under Test (SUT), specifically the application logic code. To clarify two important testing activities for the avoidance of duplicate testing effort, this paper studies possible differences between GUI testing and system testing experimentally. Specifically, we selected a SUT equipped with system tests that directly exercise the application code; We used GUITAR, a well-known GUI testing software to GUI test this SUT. Experimental results show important differences between system testing and GUI testing in terms of structural coverage and test cost

Reducing Instrumentation Overhead when Reverse-Engineering Object Interactions

Reverse-engineering object interactions from source code can be done through static, dynamic, or hybrid (static plus dynamic) analyses. In the latter two, monitoring a program and collecting runtime information translates into some overhead during program execution. Depending on the type of application, the imposed overhead can reduce the precision and accuracy of the reverse-engineered object interactions (the larger the overhead the less precise or accurate the reverse-engineered interactions), to such an extent that the reverse-engineered interactions may not be correct, especially when reverse-engineering a multithreaded software system. One is therefore seeking an instrumentation strategy as less intrusive as possible. In our past work, we showed that a hybrid approach is one step towards such a solution, compared to a purely dynamic approach, and that there is room for improvements. In this paper, we uncover, in a systematic way, other aspects of the dynamic analysis that can be improved to further reduce runtime overhead, and study alternative solutions. Our experiments show effective overhead reduction thanks to a modified procedure to collect runtime information

An orchestrated survey of available algorithms and tools for Combinatorial Testing

For functional testing based on the input domain of a functionality, parameters and their values are identified and a test suite is generated using a criterion exercising combinations of those parameters and values. Since software systems are large, resulting in large numbers of parameters and values, a technique based on combinatorics called Combinatorial Testing (CT) is used to automate the process of creating those combinations. CT is typically performed with the help of combinatorial objects called Covering Arrays. The goal of the present work is to determine available algorithms/tools for generating a combinatorial test suite. We tried to be as complete as possible by using a precise protocol for selecting papers describing those algorithms/tools. The 75 algorithms/tools we identified are then categorized on the basis of different comparison criteria, including: the test suite generation technique, the support for selection (combination) criteria, mixed covering array, the strength of coverage, and the support for constraints between parameters. Results can be of interest to researchers or software companies who are looking for a CT algorithm/tool suitable for their needs

UML diagram synthesis techniques: a systematic mapping study

Context: The Unified Modeling Language (UML), with its 14 different diagram types, is the de-facto standard modeling language for object-oriented modeling and documentation. Since the various UML diagrams describe different aspects of one, and only one, software under development, they are not independent but strongly depend on each ot her in many ways. In other words, diagrams must remain consistent. Dependencies between diagrams can become so intricate that it is sometimes even possible to synthesize one diagram on the basis of others. Support for synthesizing one UML diagram from other diagrams can provide the designer with significant help, thus speeding up the design process, decreasing the risk of errors, and guaranteeing consistency among the diagrams. Objective: The aim of this article is to provide a comprehensive summary of UML synthesis techniques as they have been described in literature to date in order to obtain an extensive and detailed overview of the current research in this area. Method: We have performed a Systematic Mapping Study by following well-known guide-lines. We selected ten primary studies by means of a

On the Effectiveness of Contracts as Test Oracles in the Detection and Diagnosis of Faults in Concurrent Object-Oriented Software

Design by Contract (DbC) is a software development methodology that focuses on clearly defining the interfaces between components to produce better quality object-oriented software. The idea behind DbC is that a method defines a contract stating the requirements a client needs to fulfill to use it, the precondition, and the properties it ensures after its execution, the postcondition. Though there exists ample support for DbC for sequential programs, applying DbC to concurrent programs presents several challenges. Using Java as the target programming language, this paper tackles such challenges by augmenting the Java Modelling Language (JML) and modifying the JML compiler to generate Runtime Assertion Checking (RAC) code to support DbC in concurrent programs. We applied our solution in a carefully designed case study on a highly concurrent industrial software system from the telecommunications domain to assess the effectiveness of contracts as test oracles in detecting and diagnosing functional faults in concurrent software. Based on these results, clear and objective requirements are defined for contracts to be effective test oracles for concurrent programs whilst balancing the effort to design them. Main results include that contracts of a realistic level of completeness and complexity can detect around 76% of faults and reduce the diagnosis effort for such faults by at least ten times. We, therefore, show that DbC can not only be applied to concurrent software but can also be a valua

A systematic identification of consistency rules for UML diagrams

UML diagrams describe different views of one piece of software. These diagrams strongly depend on each other and must therefore be consistent with one another, since inconsistencies between diagrams may be a source of faults during software development activities that rely on these diagrams. It is therefore paramount that consistency rules be defined and that inconsistencies be detected, analyzed and fixed. The relevant literature shows that authors typically define their own UML consistency rules, sometimes defining the same rules and sometimes defining rules that are already in the UML standard. The reason might be that no consolidated set of rules that are deemed relevant by authors can be found to date. The aim of our research is to provide a consolidated set of UML consistency rules and obtain a detailed overview of the current research in this area. We therefore followed a systematic procedure in order to collect and analyze UML consistency rules. We then consolidated a set of 116 UML consistency rules (avoiding redundant definitions or definitions already in the UML standard) that can be used as an important r

UML consistency rules: a systematic mapping study

Context: The Unified Modeling Language (UML), with its 14 different diagram types, is the de-facto standard tool for objectoriented modeling and documentation. Since the various UML diagrams describe different aspects of one, and only one, software under development, they are not independent but strongly depend on each other in many ways. In other words, the UML diagrams describing a software must be consistent. Inconsistencies between these diagrams may be a source of the considerable increase of faults in software systems. It is therefore paramount that these inconsistencies be detected, ana

Assessing, Comparing, and Combining Statechart- based testing and Structural testing: An Experiment

Although models have been proven to be helpful in a number of software engineering activities there is still significant resistance to model-driven development. This paper investigates one specific aspect of this larger problem. It addresses the impact of using statecharts for testing class clusters that exhibit a state-dependent behavior. More precisely, it reports on a controlled experiment that investigates their impact on testing fault-detection effectiveness. Code-based, structural testing is compared to statechart-based testing and their combination is investigated to determine whether they are complementary. Results show that there is no significant difference between the fault detection effectiveness of the two test strategies but that they are significantly more effective when combined. This implies that a cost-effective strategy would specify statechart-based test cases early on, execute them once the source code is available, and then complete them with test cases based on code coverage analysis